Light Field Factorization
Factorization is a process for decomposing data into meaningful components, or factors. One kind of data related to this invention are light fields, which describe the transport of light throughout a scene. A number of light field factorization methods are known, see U.S. Pat. No. 7,062,419 to Grzeszczuk, et al, issued Jun. 13, 2006, “Surface light field decomposition using non-negative factorization,” and U.S. Patent Application 20040249615, Grzeszczuk et al. published Dec. 9, 2004, “Surface light field decomposition using non-negative factorization.” Those methods use approximate graphical representations of objects, instead of images of real world scenes.
One factorization, method decomposes complex surface reflectance functions (spatially varying BRDFs) into a sum of products of lower dimensional (1D or 2D) components, Lawrence et al., “Inverse Shade Trees for Non-Parametric Material Representation and Editing,” ACM. Trans. on Graphics (also Proc. of ACM SIGGRAPH) (July) 2006 incorporated herein by reference. A similar method decomposes a time-varying surface appearance into a low dimensional representation that is space-time dependent, Gu et al., “Time-varying Surface Appearance: Acquisition, Modeling, and Rendering. ACM Trans, on Graphics, 2006. Both of those methods accomplish similar goals. They factorize large datasets of complex surface reflectance into terms that are compact, and at the same time, physically meaningful and editable. Because they acquire and model the full eight-dimensional BRDF, they can render under any viewing, lighting, and in the case of Gu et al, temporal condition. The primary goal of their work is to compute shade trees in computer graphic applications. However, the complexity of the BRDF acquisition makes those methods impractical for complex, outdoor scenes.
Inverse rendering measures attributes, such as lighting, textures, and the BRDF from images. Most prior art focuses on small objects and indoor scenes. One method recovers photometric properties from images of buildings, Debevec et al., “Estimating Surface Reflectance Properties of a Complex Scene under Captured Natural Illumination, USC ICT Technical Report ICT-TR-06.2004, 2004. They are able to relight and generate photo-realistic images from arbitrary viewpoints. However, their methods require measurements of the incident illumination and surface materials and a 3D model of the scene geometry. That makes the method impractical for outdoor scenes.
Another method separates the light field in a scene into direct and global components using controlled lighting, Nayar et ah, “Fast Separation of Direct and Global Components of a Scene using High Frequency Illumination,” ACM Trans. on Graphics, 2006. Obviously, it is impossible to control the lighting in outdoor scenes.
Therefore, it is desired to factor a sequence of images acquired of complex indoor or outdoor scenes into meaningful components that completely describe the scene.
Time-Lapse Photography
In time-lapse photography, a sequence of images (video) is acquired at a slow rate, and rendered at a high rate. Thus, time seems to lapse faster. Conventional time-lapse photography is often used for outdoor scenes, e.g., tidal flows, blooming flowers, and weather and traffic patterns. Time-lapse photography is also frequently used in surveillance applications.
Time-lapse photography can generate a large amount of data. For example, a single camera that takes an image every five seconds produces 17,280 images per day, or close to a million images per year. Image compression can reduce the storage requirements, but the reconstructed images typically suffer from annoying artifacts and are not very useful for further image analysis. In addition, it is difficult to edit the images in a time-lapse sequence, and advanced image-based rendering operations, such as relighting are impossible.
Therefore, a key challenge in dealing with time-lapse videos is to provide a representation that efficiently reduces storage requirements while allowing advanced image editing and useful image analysis.
One method uses intrinsic images to represent intrinsic characteristics of a scene, such as illumination, reflectance, and surface geometry, Barrow et al., “Recovering intrinsic scene characteristics from images,” Academic Press, 1978. Another method uses a maximum-likelihood framework to estimate a single reflectance image and multiple illumination images from time-lapse video, Weiss, “Deriving Intrinsic Images from Image Sequences,” IEEE International Conference on Computer Vision (ICCV), II: 68-75, 2001. That method was extended to derive time-varying reflectance and illumination images from a surveillance video, Matsushita et al., “Illumination normalization with time-dependent intrinsic images for video surveillance,” CVPR, IEEE Computer Society, 3-10, 2003.
Another method use time-lapse images to determine a reflectance field of a scene for a fixed viewpoint, Matusik et al, “Progressively-Refined Reflectance Functions from Natural Illumination,” Eurographics Symposium on Rendering, Keller et al., Eds., 299-308, 2004. They represent images as a product of the reflectance field and incident illumination. However, that method requires estimating the incident illumination using an additional light probe camera. The estimated reflectance field light combines the effects of reflectance and shadows. That method is only suitable for studio settings and not outdoor scenes.
Another method acquires image sequences with a randomly moving light source to cluster the image into regions that have similar normals, Koppal et al., “Appearance Clustering: A Novel Approach to Scene Analysis,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2006, incorporated herein by reference. The normal clusters can be used for a variety of computer vision applications, including the decomposition of the image into the terms of a linearly separable bidirectional reflectance distribution function BRDF.